{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "38032672",
   "metadata": {},
   "source": [
    "# Batch inference and drift detection\n",
    "\n",
    "This tutorial leverages a function from the [MLRun Function Hub](https://www.mlrun.org/hub/) to perform [batch inference](https://www.mlrun.org/hub/functions/master/batch_inference/) using a logged model and a new prediction dataset. The function also calculates data drift by comparing the new prediction dataset with the original training set.\n",
    "\n",
    "Make sure you have reviewed the basics in MLRun [**Quick Start Tutorial**](../01-mlrun-basics.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "594b584a",
   "metadata": {},
   "source": [
    "Tutorial steps:\n",
    "- [**Set up an MLRun project**](#setup-project)\n",
    "- [**View the data**](#view-data)\n",
    "- [**Log a model with a given framework and training set**](#log-model)\n",
    "- [**Import and run the batch inference function**](#run)\n",
    "- [**View predictions and drift status**](#view-results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d886a1e0",
   "metadata": {},
   "source": [
    "## MLRun installation and Configuration\n",
    "\n",
    "Before running this notebook make sure `mlrun` is installed and that you have configured the access to the MLRun service. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84529cbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install MLRun if not installed, run this only once (restart the notebook after the install !!!)\n",
    "%pip install mlrun"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4a91c44",
   "metadata": {},
   "source": [
    "<a id=\"setup-project\"></a>\n",
    "## Set up a project\n",
    "\n",
    "First, import the dependencies and create an [MLRun project](https://docs.mlrun.org/en/latest/projects/project.html). The project contains all of your models, functions, datasets, etc.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5604fe9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlrun\n",
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5305421b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-03-15 10:11:49,387 [info] loaded project tutorial from MLRun DB\n"
     ]
    }
   ],
   "source": [
    "project = mlrun.get_or_create_project(\"tutorial\", context=\"./\", user_project=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f9cf4bf",
   "metadata": {},
   "source": [
    "```{admonition} Note\n",
    "This tutorial does not focus on training a model. Instead, it starts with a trained model and its corresponding training and prediction dataset.\n",
    "```\n",
    "\n",
    "You will use the following model files and datasets to perform the batch prediction. The model is a [DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) from sklearn and the datasets are in `parquet` format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bbccfbd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We choose the correct model to avoid pickle warnings\n",
    "import sys\n",
    "\n",
    "suffix = (\n",
    "    mlrun.__version__.split(\"-\")[0].replace(\".\", \"_\")\n",
    "    if sys.version_info[1] > 7\n",
    "    else \"3.7\"\n",
    ")\n",
    "\n",
    "model_path = mlrun.get_sample_path(f\"models/batch-predict/model-{suffix}.pkl\")\n",
    "training_set_path = mlrun.get_sample_path(\"data/batch-predict/training_set.parquet\")\n",
    "prediction_set_path = mlrun.get_sample_path(\"data/batch-predict/prediction_set.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c45d314",
   "metadata": {},
   "source": [
    "<a id=\"view-data\"></a>\n",
    "## View the data\n",
    "\n",
    "The training data has 20 numerical features and a binary (0,1) label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6d0a46d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature_0</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>feature_5</th>\n",
       "      <th>feature_6</th>\n",
       "      <th>feature_7</th>\n",
       "      <th>feature_8</th>\n",
       "      <th>feature_9</th>\n",
       "      <th>...</th>\n",
       "      <th>feature_11</th>\n",
       "      <th>feature_12</th>\n",
       "      <th>feature_13</th>\n",
       "      <th>feature_14</th>\n",
       "      <th>feature_15</th>\n",
       "      <th>feature_16</th>\n",
       "      <th>feature_17</th>\n",
       "      <th>feature_18</th>\n",
       "      <th>feature_19</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.572754</td>\n",
       "      <td>0.171079</td>\n",
       "      <td>0.403080</td>\n",
       "      <td>0.955429</td>\n",
       "      <td>0.272039</td>\n",
       "      <td>0.360277</td>\n",
       "      <td>-0.995429</td>\n",
       "      <td>0.437239</td>\n",
       "      <td>0.991556</td>\n",
       "      <td>0.010004</td>\n",
       "      <td>...</td>\n",
       "      <td>0.112194</td>\n",
       "      <td>-0.319256</td>\n",
       "      <td>-0.392631</td>\n",
       "      <td>-0.290766</td>\n",
       "      <td>1.265054</td>\n",
       "      <td>1.037082</td>\n",
       "      <td>-1.200076</td>\n",
       "      <td>0.820992</td>\n",
       "      <td>0.834868</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.623733</td>\n",
       "      <td>-0.149823</td>\n",
       "      <td>-1.410537</td>\n",
       "      <td>-0.729388</td>\n",
       "      <td>-1.996337</td>\n",
       "      <td>-1.213348</td>\n",
       "      <td>1.461307</td>\n",
       "      <td>1.187854</td>\n",
       "      <td>-1.790926</td>\n",
       "      <td>-0.981600</td>\n",
       "      <td>...</td>\n",
       "      <td>0.428653</td>\n",
       "      <td>-0.503820</td>\n",
       "      <td>-0.798035</td>\n",
       "      <td>2.038105</td>\n",
       "      <td>-3.080463</td>\n",
       "      <td>0.408561</td>\n",
       "      <td>1.647116</td>\n",
       "      <td>-0.838553</td>\n",
       "      <td>0.680983</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.814168</td>\n",
       "      <td>-0.221412</td>\n",
       "      <td>0.020822</td>\n",
       "      <td>1.066718</td>\n",
       "      <td>-0.573164</td>\n",
       "      <td>0.067838</td>\n",
       "      <td>0.923045</td>\n",
       "      <td>0.338146</td>\n",
       "      <td>0.981413</td>\n",
       "      <td>1.481757</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.052559</td>\n",
       "      <td>-0.241873</td>\n",
       "      <td>-1.232272</td>\n",
       "      <td>-0.010758</td>\n",
       "      <td>0.806800</td>\n",
       "      <td>0.661162</td>\n",
       "      <td>0.589018</td>\n",
       "      <td>0.522137</td>\n",
       "      <td>-0.924624</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.062279</td>\n",
       "      <td>-0.966309</td>\n",
       "      <td>0.341471</td>\n",
       "      <td>-0.737059</td>\n",
       "      <td>1.460671</td>\n",
       "      <td>0.367851</td>\n",
       "      <td>-0.435336</td>\n",
       "      <td>0.445308</td>\n",
       "      <td>-0.655663</td>\n",
       "      <td>-0.196220</td>\n",
       "      <td>...</td>\n",
       "      <td>0.641017</td>\n",
       "      <td>0.099059</td>\n",
       "      <td>1.902592</td>\n",
       "      <td>-1.024929</td>\n",
       "      <td>0.030703</td>\n",
       "      <td>-0.198751</td>\n",
       "      <td>-0.342009</td>\n",
       "      <td>-1.286865</td>\n",
       "      <td>-1.118373</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.195755</td>\n",
       "      <td>0.576332</td>\n",
       "      <td>-0.260496</td>\n",
       "      <td>0.841489</td>\n",
       "      <td>0.398269</td>\n",
       "      <td>-0.717972</td>\n",
       "      <td>0.810550</td>\n",
       "      <td>-1.058326</td>\n",
       "      <td>0.368610</td>\n",
       "      <td>0.606007</td>\n",
       "      <td>...</td>\n",
       "      <td>0.195267</td>\n",
       "      <td>0.876144</td>\n",
       "      <td>0.151615</td>\n",
       "      <td>0.094867</td>\n",
       "      <td>0.627353</td>\n",
       "      <td>-0.389023</td>\n",
       "      <td>0.662846</td>\n",
       "      <td>-0.857000</td>\n",
       "      <td>1.091218</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   feature_0  feature_1  feature_2  feature_3  feature_4  feature_5  \\\n",
       "0   0.572754   0.171079   0.403080   0.955429   0.272039   0.360277   \n",
       "1   0.623733  -0.149823  -1.410537  -0.729388  -1.996337  -1.213348   \n",
       "2   0.814168  -0.221412   0.020822   1.066718  -0.573164   0.067838   \n",
       "3   1.062279  -0.966309   0.341471  -0.737059   1.460671   0.367851   \n",
       "4   0.195755   0.576332  -0.260496   0.841489   0.398269  -0.717972   \n",
       "\n",
       "   feature_6  feature_7  feature_8  feature_9  ...  feature_11  feature_12  \\\n",
       "0  -0.995429   0.437239   0.991556   0.010004  ...    0.112194   -0.319256   \n",
       "1   1.461307   1.187854  -1.790926  -0.981600  ...    0.428653   -0.503820   \n",
       "2   0.923045   0.338146   0.981413   1.481757  ...   -1.052559   -0.241873   \n",
       "3  -0.435336   0.445308  -0.655663  -0.196220  ...    0.641017    0.099059   \n",
       "4   0.810550  -1.058326   0.368610   0.606007  ...    0.195267    0.876144   \n",
       "\n",
       "   feature_13  feature_14  feature_15  feature_16  feature_17  feature_18  \\\n",
       "0   -0.392631   -0.290766    1.265054    1.037082   -1.200076    0.820992   \n",
       "1   -0.798035    2.038105   -3.080463    0.408561    1.647116   -0.838553   \n",
       "2   -1.232272   -0.010758    0.806800    0.661162    0.589018    0.522137   \n",
       "3    1.902592   -1.024929    0.030703   -0.198751   -0.342009   -1.286865   \n",
       "4    0.151615    0.094867    0.627353   -0.389023    0.662846   -0.857000   \n",
       "\n",
       "   feature_19  label  \n",
       "0    0.834868      0  \n",
       "1    0.680983      1  \n",
       "2   -0.924624      0  \n",
       "3   -1.118373      1  \n",
       "4    1.091218      1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_parquet(training_set_path).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5d924d9",
   "metadata": {},
   "source": [
    "**The prediciton data has 20 numerical features, but no label - this is what you will predict:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "522c2eff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature_0</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>feature_5</th>\n",
       "      <th>feature_6</th>\n",
       "      <th>feature_7</th>\n",
       "      <th>feature_8</th>\n",
       "      <th>feature_9</th>\n",
       "      <th>feature_10</th>\n",
       "      <th>feature_11</th>\n",
       "      <th>feature_12</th>\n",
       "      <th>feature_13</th>\n",
       "      <th>feature_14</th>\n",
       "      <th>feature_15</th>\n",
       "      <th>feature_16</th>\n",
       "      <th>feature_17</th>\n",
       "      <th>feature_18</th>\n",
       "      <th>feature_19</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.059506</td>\n",
       "      <td>-1.314291</td>\n",
       "      <td>2.721516</td>\n",
       "      <td>-2.132869</td>\n",
       "      <td>-0.693963</td>\n",
       "      <td>0.376643</td>\n",
       "      <td>3.017790</td>\n",
       "      <td>3.876329</td>\n",
       "      <td>-1.294736</td>\n",
       "      <td>0.030773</td>\n",
       "      <td>0.401491</td>\n",
       "      <td>2.775699</td>\n",
       "      <td>2.361580</td>\n",
       "      <td>0.173441</td>\n",
       "      <td>0.879510</td>\n",
       "      <td>1.141007</td>\n",
       "      <td>4.608280</td>\n",
       "      <td>-0.518388</td>\n",
       "      <td>0.129690</td>\n",
       "      <td>2.794967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.190382</td>\n",
       "      <td>0.891571</td>\n",
       "      <td>3.726070</td>\n",
       "      <td>0.673870</td>\n",
       "      <td>-0.252565</td>\n",
       "      <td>-0.729156</td>\n",
       "      <td>2.646563</td>\n",
       "      <td>4.782729</td>\n",
       "      <td>0.318952</td>\n",
       "      <td>-0.781567</td>\n",
       "      <td>1.473632</td>\n",
       "      <td>1.101721</td>\n",
       "      <td>3.723400</td>\n",
       "      <td>-0.466867</td>\n",
       "      <td>-0.056224</td>\n",
       "      <td>3.344701</td>\n",
       "      <td>0.194332</td>\n",
       "      <td>0.463992</td>\n",
       "      <td>0.292268</td>\n",
       "      <td>4.665876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.996384</td>\n",
       "      <td>-0.099537</td>\n",
       "      <td>3.421476</td>\n",
       "      <td>0.162771</td>\n",
       "      <td>-1.143458</td>\n",
       "      <td>-1.026791</td>\n",
       "      <td>2.114702</td>\n",
       "      <td>2.517553</td>\n",
       "      <td>-0.154620</td>\n",
       "      <td>-0.465423</td>\n",
       "      <td>-1.723025</td>\n",
       "      <td>1.729386</td>\n",
       "      <td>2.820340</td>\n",
       "      <td>-1.041428</td>\n",
       "      <td>-0.331871</td>\n",
       "      <td>2.909172</td>\n",
       "      <td>2.138613</td>\n",
       "      <td>-0.046252</td>\n",
       "      <td>-0.732631</td>\n",
       "      <td>4.716266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.289976</td>\n",
       "      <td>-1.680019</td>\n",
       "      <td>3.126478</td>\n",
       "      <td>-0.704451</td>\n",
       "      <td>-1.149112</td>\n",
       "      <td>1.174962</td>\n",
       "      <td>2.860341</td>\n",
       "      <td>3.753661</td>\n",
       "      <td>-0.326119</td>\n",
       "      <td>2.128411</td>\n",
       "      <td>-0.508000</td>\n",
       "      <td>2.328688</td>\n",
       "      <td>3.397321</td>\n",
       "      <td>-0.932060</td>\n",
       "      <td>-1.442370</td>\n",
       "      <td>2.058517</td>\n",
       "      <td>3.881936</td>\n",
       "      <td>2.090635</td>\n",
       "      <td>-0.045832</td>\n",
       "      <td>4.197315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.294866</td>\n",
       "      <td>1.044919</td>\n",
       "      <td>2.924139</td>\n",
       "      <td>0.814049</td>\n",
       "      <td>-1.455054</td>\n",
       "      <td>-0.270432</td>\n",
       "      <td>3.380195</td>\n",
       "      <td>2.339669</td>\n",
       "      <td>1.029101</td>\n",
       "      <td>-1.171018</td>\n",
       "      <td>-1.459395</td>\n",
       "      <td>1.283565</td>\n",
       "      <td>0.677006</td>\n",
       "      <td>-2.147444</td>\n",
       "      <td>-0.494150</td>\n",
       "      <td>3.222041</td>\n",
       "      <td>6.219348</td>\n",
       "      <td>-1.914110</td>\n",
       "      <td>0.317786</td>\n",
       "      <td>4.143443</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   feature_0  feature_1  feature_2  feature_3  feature_4  feature_5  \\\n",
       "0  -2.059506  -1.314291   2.721516  -2.132869  -0.693963   0.376643   \n",
       "1  -1.190382   0.891571   3.726070   0.673870  -0.252565  -0.729156   \n",
       "2  -0.996384  -0.099537   3.421476   0.162771  -1.143458  -1.026791   \n",
       "3  -0.289976  -1.680019   3.126478  -0.704451  -1.149112   1.174962   \n",
       "4  -0.294866   1.044919   2.924139   0.814049  -1.455054  -0.270432   \n",
       "\n",
       "   feature_6  feature_7  feature_8  feature_9  feature_10  feature_11  \\\n",
       "0   3.017790   3.876329  -1.294736   0.030773    0.401491    2.775699   \n",
       "1   2.646563   4.782729   0.318952  -0.781567    1.473632    1.101721   \n",
       "2   2.114702   2.517553  -0.154620  -0.465423   -1.723025    1.729386   \n",
       "3   2.860341   3.753661  -0.326119   2.128411   -0.508000    2.328688   \n",
       "4   3.380195   2.339669   1.029101  -1.171018   -1.459395    1.283565   \n",
       "\n",
       "   feature_12  feature_13  feature_14  feature_15  feature_16  feature_17  \\\n",
       "0    2.361580    0.173441    0.879510    1.141007    4.608280   -0.518388   \n",
       "1    3.723400   -0.466867   -0.056224    3.344701    0.194332    0.463992   \n",
       "2    2.820340   -1.041428   -0.331871    2.909172    2.138613   -0.046252   \n",
       "3    3.397321   -0.932060   -1.442370    2.058517    3.881936    2.090635   \n",
       "4    0.677006   -2.147444   -0.494150    3.222041    6.219348   -1.914110   \n",
       "\n",
       "   feature_18  feature_19  \n",
       "0    0.129690    2.794967  \n",
       "1    0.292268    4.665876  \n",
       "2   -0.732631    4.716266  \n",
       "3   -0.045832    4.197315  \n",
       "4    0.317786    4.143443  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_parquet(prediction_set_path).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "892b09bc",
   "metadata": {},
   "source": [
    "<a id=\"log-model\"></a>\n",
    "## Log the model with training data\n",
    "\n",
    "Next, log the model using MLRun experiment tracking. This is usually done in a training pipeline, but you can also bring in your pre-trained models from other sources. See [Working with data and model artifacts](https://docs.mlrun.org/en/latest/training/working-with-data-and-model-artifacts.html) and [Automated experiment tracking](https://docs.mlrun.org/en/latest/concepts/auto-logging-mlops.html) for more information.\n",
    "\n",
    "In this example, you are logging a training set with the model for future comparison, however you can also directly pass in your training set to the batch prediction function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d8e0ab34",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_artifact = project.log_model(\n",
    "    key=\"model\",\n",
    "    model_file=model_path,\n",
    "    framework=\"sklearn\",\n",
    "    training_set=pd.read_parquet(training_set_path),\n",
    "    label_column=\"label\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c681527",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the model artifact unique URI\n",
    "model_artifact.uri"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f1b9f7a",
   "metadata": {},
   "source": [
    "<a id=\"run\"></a>\n",
    "## Import and run the batch inference function\n",
    "\n",
    "Next, import the [batch inference](https://www.mlrun.org/hub/functions/master/batch_inference/) function from the [MLRun Function Hub](https://www.mlrun.org/hub/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "040650f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "fn = mlrun.import_function(\"hub://batch_inference\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5f44ada",
   "metadata": {},
   "source": [
    "### Run batch inference\n",
    "\n",
    "Finally, perform the batch prediction by passing in your model and datasets. See the corresponding [batch inference example notebook](https://github.com/mlrun/functions/blob/development/batch_inference/batch_inference.ipynb) for an exhaustive list of other parameters that are supported:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "233afac5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-03-15 10:11:50,578 [info] starting run batch-inference-infer uid=b357c4bc6ccf48e8a18803bab02f919a DB=http://mlrun-api:8080\n",
      "> 2023-03-15 10:11:50,802 [info] Job is running in the background, pod: batch-inference-infer-wzdcg\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>tutorial-yonis</td>\n",
       "      <td><div title=\"b357c4bc6ccf48e8a18803bab02f919a\"><a href=\"https://dashboard.default-tenant.app.cto-office.iguazio-cd1.com/mlprojects/tutorial-yonis/jobs/monitor/b357c4bc6ccf48e8a18803bab02f919a/overview\" target=\"_blank\" >...b02f919a</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Mar 15 10:11:54</td>\n",
       "      <td>completed</td>\n",
       "      <td>batch-inference-infer</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=yonis</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=yonis</div><div class=\"dictlist\">mlrun/client_version=1.2.1</div><div class=\"dictlist\">host=batch-inference-infer-wzdcg</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/batch-predict/prediction_set.parquet\">dataset</div></td>\n",
       "      <td><div class=\"dictlist\">model=store://models/tutorial-yonis/model#0:69f79f68-b455-45d8-a461-9340fb64fb28</div><div class=\"dictlist\">perform_drift_analysis=True</div></td>\n",
       "      <td><div class=\"dictlist\">batch_id=8616574bd1078ebdd43d2bf350d8d14321d2072dd969ebccc8c55afd</div><div class=\"dictlist\">drift_status=False</div><div class=\"dictlist\">drift_metric=0.31451973312099435</div></td>\n",
       "      <td><div title=\"v3io:///projects/tutorial-yonis/artifacts/batch-inference-infer/0/prediction.parquet\">prediction</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result31a6fbf6\" title=\"files/v3io/projects/tutorial-yonis/artifacts/batch-inference-infer/0/drift_table_plot.html\">drift_table_plot</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result31a6fbf6\" title=\"files/v3io/projects/tutorial-yonis/artifacts/batch-inference-infer/0/features_drift_results.json\">features_drift_results</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result31a6fbf6-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result31a6fbf6-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result31a6fbf6\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result31a6fbf6-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cto-office.iguazio-cd1.com/mlprojects/tutorial-yonis/jobs/monitor/b357c4bc6ccf48e8a18803bab02f919a/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-03-15 10:12:03,958 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "run = project.run_function(\n",
    "    fn,\n",
    "    inputs={\n",
    "        \"dataset\": prediction_set_path,\n",
    "        # If you do not log a dataset with your model, you can pass it in here:\n",
    "        #       \"sample_set\" : training_set_path\n",
    "    },\n",
    "    params={\n",
    "        \"model\": model_artifact.uri,\n",
    "        \"perform_drift_analysis\": True,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45417d3c",
   "metadata": {},
   "source": [
    "<a id=\"view-results\"></a>\n",
    "## View predictions and drift status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5434192c",
   "metadata": {},
   "source": [
    "These are the batch predictions on the prediction set from the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1c651b0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature_0</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>feature_4</th>\n",
       "      <th>feature_5</th>\n",
       "      <th>feature_6</th>\n",
       "      <th>feature_7</th>\n",
       "      <th>feature_8</th>\n",
       "      <th>feature_9</th>\n",
       "      <th>...</th>\n",
       "      <th>feature_11</th>\n",
       "      <th>feature_12</th>\n",
       "      <th>feature_13</th>\n",
       "      <th>feature_14</th>\n",
       "      <th>feature_15</th>\n",
       "      <th>feature_16</th>\n",
       "      <th>feature_17</th>\n",
       "      <th>feature_18</th>\n",
       "      <th>feature_19</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.059506</td>\n",
       "      <td>-1.314291</td>\n",
       "      <td>2.721516</td>\n",
       "      <td>-2.132869</td>\n",
       "      <td>-0.693963</td>\n",
       "      <td>0.376643</td>\n",
       "      <td>3.017790</td>\n",
       "      <td>3.876329</td>\n",
       "      <td>-1.294736</td>\n",
       "      <td>0.030773</td>\n",
       "      <td>...</td>\n",
       "      <td>2.775699</td>\n",
       "      <td>2.361580</td>\n",
       "      <td>0.173441</td>\n",
       "      <td>0.879510</td>\n",
       "      <td>1.141007</td>\n",
       "      <td>4.608280</td>\n",
       "      <td>-0.518388</td>\n",
       "      <td>0.129690</td>\n",
       "      <td>2.794967</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.190382</td>\n",
       "      <td>0.891571</td>\n",
       "      <td>3.726070</td>\n",
       "      <td>0.673870</td>\n",
       "      <td>-0.252565</td>\n",
       "      <td>-0.729156</td>\n",
       "      <td>2.646563</td>\n",
       "      <td>4.782729</td>\n",
       "      <td>0.318952</td>\n",
       "      <td>-0.781567</td>\n",
       "      <td>...</td>\n",
       "      <td>1.101721</td>\n",
       "      <td>3.723400</td>\n",
       "      <td>-0.466867</td>\n",
       "      <td>-0.056224</td>\n",
       "      <td>3.344701</td>\n",
       "      <td>0.194332</td>\n",
       "      <td>0.463992</td>\n",
       "      <td>0.292268</td>\n",
       "      <td>4.665876</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.996384</td>\n",
       "      <td>-0.099537</td>\n",
       "      <td>3.421476</td>\n",
       "      <td>0.162771</td>\n",
       "      <td>-1.143458</td>\n",
       "      <td>-1.026791</td>\n",
       "      <td>2.114702</td>\n",
       "      <td>2.517553</td>\n",
       "      <td>-0.154620</td>\n",
       "      <td>-0.465423</td>\n",
       "      <td>...</td>\n",
       "      <td>1.729386</td>\n",
       "      <td>2.820340</td>\n",
       "      <td>-1.041428</td>\n",
       "      <td>-0.331871</td>\n",
       "      <td>2.909172</td>\n",
       "      <td>2.138613</td>\n",
       "      <td>-0.046252</td>\n",
       "      <td>-0.732631</td>\n",
       "      <td>4.716266</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.289976</td>\n",
       "      <td>-1.680019</td>\n",
       "      <td>3.126478</td>\n",
       "      <td>-0.704451</td>\n",
       "      <td>-1.149112</td>\n",
       "      <td>1.174962</td>\n",
       "      <td>2.860341</td>\n",
       "      <td>3.753661</td>\n",
       "      <td>-0.326119</td>\n",
       "      <td>2.128411</td>\n",
       "      <td>...</td>\n",
       "      <td>2.328688</td>\n",
       "      <td>3.397321</td>\n",
       "      <td>-0.932060</td>\n",
       "      <td>-1.442370</td>\n",
       "      <td>2.058517</td>\n",
       "      <td>3.881936</td>\n",
       "      <td>2.090635</td>\n",
       "      <td>-0.045832</td>\n",
       "      <td>4.197315</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.294866</td>\n",
       "      <td>1.044919</td>\n",
       "      <td>2.924139</td>\n",
       "      <td>0.814049</td>\n",
       "      <td>-1.455054</td>\n",
       "      <td>-0.270432</td>\n",
       "      <td>3.380195</td>\n",
       "      <td>2.339669</td>\n",
       "      <td>1.029101</td>\n",
       "      <td>-1.171018</td>\n",
       "      <td>...</td>\n",
       "      <td>1.283565</td>\n",
       "      <td>0.677006</td>\n",
       "      <td>-2.147444</td>\n",
       "      <td>-0.494150</td>\n",
       "      <td>3.222041</td>\n",
       "      <td>6.219348</td>\n",
       "      <td>-1.914110</td>\n",
       "      <td>0.317786</td>\n",
       "      <td>4.143443</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   feature_0  feature_1  feature_2  feature_3  feature_4  feature_5  \\\n",
       "0  -2.059506  -1.314291   2.721516  -2.132869  -0.693963   0.376643   \n",
       "1  -1.190382   0.891571   3.726070   0.673870  -0.252565  -0.729156   \n",
       "2  -0.996384  -0.099537   3.421476   0.162771  -1.143458  -1.026791   \n",
       "3  -0.289976  -1.680019   3.126478  -0.704451  -1.149112   1.174962   \n",
       "4  -0.294866   1.044919   2.924139   0.814049  -1.455054  -0.270432   \n",
       "\n",
       "   feature_6  feature_7  feature_8  feature_9  ...  feature_11  feature_12  \\\n",
       "0   3.017790   3.876329  -1.294736   0.030773  ...    2.775699    2.361580   \n",
       "1   2.646563   4.782729   0.318952  -0.781567  ...    1.101721    3.723400   \n",
       "2   2.114702   2.517553  -0.154620  -0.465423  ...    1.729386    2.820340   \n",
       "3   2.860341   3.753661  -0.326119   2.128411  ...    2.328688    3.397321   \n",
       "4   3.380195   2.339669   1.029101  -1.171018  ...    1.283565    0.677006   \n",
       "\n",
       "   feature_13  feature_14  feature_15  feature_16  feature_17  feature_18  \\\n",
       "0    0.173441    0.879510    1.141007    4.608280   -0.518388    0.129690   \n",
       "1   -0.466867   -0.056224    3.344701    0.194332    0.463992    0.292268   \n",
       "2   -1.041428   -0.331871    2.909172    2.138613   -0.046252   -0.732631   \n",
       "3   -0.932060   -1.442370    2.058517    3.881936    2.090635   -0.045832   \n",
       "4   -2.147444   -0.494150    3.222041    6.219348   -1.914110    0.317786   \n",
       "\n",
       "   feature_19  label  \n",
       "0    2.794967      0  \n",
       "1    4.665876      1  \n",
       "2    4.716266      0  \n",
       "3    4.197315      0  \n",
       "4    4.143443      0  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run.artifact(\"prediction\").as_df().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b1fe08a",
   "metadata": {},
   "source": [
    "There is also a drift table plot that compares the drift between the training data and prediction data per feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db45da66",
   "metadata": {},
   "outputs": [],
   "source": [
    "run.artifact(\"drift_table_plot\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a1bf631",
   "metadata": {},
   "source": [
    "![drift_table_plot](../_static/images/tutorial/drift_table_plot.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8606a61b",
   "metadata": {},
   "source": [
    "Finally, you also get a numerical drift metric and boolean flag denoting whether or not data drift is detected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2a787831",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_id': '8616574bd1078ebdd43d2bf350d8d14321d2072dd969ebccc8c55afd',\n",
       " 'drift_status': False,\n",
       " 'drift_metric': 0.31451973312099435}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run.status.results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3b2663c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'feature_14': 0.046364723781764774,\n",
       " 'feature_10': 0.042567035578799796,\n",
       " 'feature_1': 0.04485072701663093,\n",
       " 'feature_2': 0.7391279921664593,\n",
       " 'feature_0': 0.028086840976606773,\n",
       " 'feature_17': 0.03587785749574268,\n",
       " 'feature_8': 0.039060131873550404,\n",
       " 'feature_13': 0.04239724655474124,\n",
       " 'feature_15': 0.6329075683793959,\n",
       " 'feature_19': 0.7902698698155215,\n",
       " 'feature_9': 0.04468363504674985,\n",
       " 'feature_16': 0.7181622588902428,\n",
       " 'label': 0.33613674196785814,\n",
       " 'feature_5': 0.05184219833790496,\n",
       " 'feature_12': 0.7034787615778625,\n",
       " 'feature_3': 0.043769819014849734,\n",
       " 'feature_18': 0.04443732609382538,\n",
       " 'feature_6': 0.7262042202197605,\n",
       " 'feature_7': 0.7297906294873706,\n",
       " 'feature_11': 0.7221431701127441,\n",
       " 'feature_4': 0.042755641152500176}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data/concept drift per feature\n",
    "import json\n",
    "\n",
    "json.loads(run.artifact(\"features_drift_results\").get())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf3d9159",
   "metadata": {},
   "source": [
    "## Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abaf6776",
   "metadata": {},
   "source": [
    "In a production setting, you probably want to incorporate this as part of a larger pipeline or application.\n",
    "\n",
    "For example, if you use this function for the prediction capabilities, you can pass the `prediction` output as the input to another pipeline step, store it in an external location like S3, or send to an application or user.\n",
    "\n",
    "If you use this function for the drift detection capabilities, you can use the `drift_status` and `drift_metrics` outputs to automate further pipeline steps, send a notification, or kick off a re-training pipeline."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
